1 IMAGE ENHANCEMENT USING FACE DETECTION 

2 Technical Field 

3 The technical field relates to image enhancement, and, in particular, to image 

4 enhancement using face detection. 

5 Background 

6 Appearances of faces in images have strong impact on how the images are perceived. 

7 Since many images are acquired with faces too bright or too dark, or with a red eye artifact 

8 resulting from flashes, image enhancement techniques are becoming increasingly important. 

9 Traditional methods for image enhancement typically work by modifying lightness, 



1 0 contrast, or color levels to improve image appearance. However, such methods typically work 

rl 1 using only lower-level image attributes. For example, the well-known method of histogram 

|32 equalization uses only image histogram. Moreover, such traditional methods may require human 

^2 3 involvement during and as part of the image enhancement process, with the human controlling 

Ul4 the levels of modification. 



jrt 5 Traditional red eye removal techniques typically require a user to click on or near eyes 

1| 6 in an image that exhibit the red eye artifact, in other words, user interaction is typically required. 

U17 Summary 

Iff 8 An image enhancement method using face detection provides for automatic detection of 

r}9 human faces in an image using face detection algorithms and automatic enhancement of 

20 appearances of the image based on knowledge of faces in the image. 

21 In an embodiment, the image enhancement method may automatically enhance lightness, 

22 contrast, or color levels of the human faces. 

23 In another embodiment, the image enhancement method may automatically locate the 

24 human faces in the image, locate eyes in the human faces, and reduce or remove any red eye 

25 artifact from the human faces. 

26 In yet another embodiment, the image enhancement method may use mapping techniques 

27 to produce an image with target levels for a mean value and/or a variation value, such as a 

28 standard deviation, in the face regions. The mapping may modify the faces alone or may modify 

29 the entire image. 
30 
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1 Description of the Drawings 

2 The preferred embodiments of an image enhancement method using face detection will 

3 be described in detail with reference to the following figures, in which like numerals refer to like 

4 elements, and wherein: 

5 Figure 1 illustrates exemplary hardware components of a computer that may be used to 

6 implement the image enhancement method using face detection; 

7 Figure 2(a) illustrates a first exemplary image enhancement method using lightness 

8 mapping; 

9 Figure 2(b) illustrates a second exemplary image enhancement method using lightness 
10 mapping; and 

0 1 Figure 3 is a flow chart of an exemplary image enhancement method using face detection. 

0§2 Detailed Description 

113 An image enhancement apparatus and a corresponding method use face detection to 



yj4 provide for automatic enhancement of appearances of an image based on knowledge of human 

f|5 faces in the image. By modifying and transforming the image automatically using facial 

^46 information, the image, including the human faces in the image, may have more pleasing 

fl 7 lightness, contrast, and/or color levels. The image enhancement method may also automatically 

yf 8 reduce or remove any red eye artifact without human intervention, leading to images with more 

rj 9 pleasing appearances . 

20 Figure 1 illustrates exemplary hardware components of a computer 1 00 that may be used 

2 1 to implement the image enhancement method using face detection. The computer 1 00 includes 

22 a connection with a network 118 such as the Internet or other type of computer or phone 

23 networks. The computer 1 00 typically includes a memory 1 02, a secondary storage device 112, 

24 a processor 1 14, an input device 1 1 6, a display device 1 1 0, and an output device 1 08. 

25 The memory 102 may include random access memory (RAM) or similar types of 

26 memory. The computer 1 00 may be connected to the network 1 1 8 by a web browser. The web 

27 browser makes a connection via the WWW to other computers known as web servers, and 

28 receives information from the web servers that is displayed on the computer 1 00, The secondary 

29 storage device 1 12 may include a hard disk drive, floppy disk drive, CD-ROM drive, or other 

30 types of non-volatile data storage, and may correspond with various databases or other resources. 



HP No. 1 0006299- 1 



1 The processor 114 may execute information stored in the memory 102, the secondary storage 

2 1 12, or received from the Internet or other network 118. The input device 1 16 may include any 

3 device for entering data into the computer 100, such as a keyboard, key pad, cursor-control 

4 device, touch-screen (possibly with a stylus), or microphone. The display device 110 may 

5 include any type of device for presenting visual image, such as, for example, a computer monitor, 

6 flat-screen display, or display panel. The output device 108 may include any type of device for 

7 presenting data in hard copy format, such as a printer, and other types of output devices including 

8 speakers or any device for providing data in audio form. The computer 1 00 can possibly include 

9 multiple input devices, output devices, and display devices. 

10 Although the computer 100 is depicted with various components, one skilled in the art 

Cll will appreciate that the computer 100 can contain additional or different components. In 

||2 addition, although aspects of an implementation consistent with the present invention are 

s -|3 described as being stored in memory, one skilled in the art will appreciate that these aspects can 

y|4 also be stored on or read from other types of computer program products or computer-readable 

S5 media, such as secondary storage devices, including hard disks, floppy disks, or CD-ROM; a 

^4 6 carrier wave from the Internet or other network; or other forms of RAM or ROM. The computer- 

fl 7 readable media may include instructions for controlling the computer 1 00 to perform a particular 

||| 8 method. 

3 9 After an image, such as a photograph or a digital image, is inputted into the memory 1 02 

20 through the input device 1 1 6, the secondary storage 1 1 2, or other means, the processor 1 1 4 may 

21 automatically detect and locate faces, typically human faces, in the image using face detection 

22 algorithms. Human faces have distinctive appearances, and the face detection algorithms 

23 typically use lightness information to detect and locate faces in an image by extracting out a 

24 lightness version of the image. The processor 1 14 may further locate components of the faces, 

25 such as eyes. The automatic location of eyes in the faces may enable automatic red eye reduction 

26 or removal (described later). 

27 Examples of the face detection algorithms are described, for example, in Rowley, Baluj a, 

28 and Kanade, "Neural Network-Based Face Detection," IEEE Transactions on Pattern Analysis 

29 and Machine Intelligence, Vol. 20, No. 1, January 1998; Sung and Poggio, "Example-Based 

30 Learning for View-Based Human Face Detection," IEEE Transactions on Pattern Analysis and 
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1 Machine Intelligence, Vol. 20, No. 1, January 1998; and U.S. Patent No. 5,642,431, issued to 

2 Poggio and Sung, entitled "Network-Based System and Method for Detection of Faces and the 

3 Like", which are incorporated herein by reference. 

4 "Neural Network-B ased Face Detection" presents a neural network-based face detection 

5 system. A retinally connected neural network examines small windows of an image, and decides 

6 whether each window contains a face. The system arbitrates among multiple networks to 

7 improve performance over a single network. A bootstrap algorithm is used for training the 

8 networks, which adds false detections into the training set as training progresses . This eliminates 

9 the difficult task of manually selecting non-face training examples, which must be chosen to span 
10 the entire space of non-face images. 

H 1 "Example-Based Learning for View-Based Human Face Detection" presents an example- 

^2 based learning approach for locating vertical frontal views of human faces in complex scenes. 

y|3 The technique models the distribution of human face patterns by means of a few view-based 

y|4 "face" and "nonface" model clusters. At each image location, a difference feature vector is 

g 5 computed between the local image pattern and the distribution-based model. A trained classifier 

y 6 determines, based on the difference feature vector measurements, whether or not a human face 

III 7 exists at the current image location. The article shows empirically that a distance metric adopted 

yj 8 for computing difference feature vectors, and the "nonface" clusters included in the distribution- 

M9 based model, are both critical for the success of the system. 

20 U.S. Patent No. 5,642,43 1 discloses a network-based system and method for analyzing 

21 images to detect human faces using a trained neural network. Because human faces are 

22 essentially structured objects with the same key features geometrically arranged in roughly the 

23 same fashion, U.S. Patent No. 5,642,43 ldefines a semantically stable "canonical" face pattern 

24 in the image domain for the purpose of pattern matching. 

25 As an example, the processor 1 1 4 may detect human faces by scanning an image for such 

26 canonical face-like patterns at all possible scales. The scales represent how coarsely the image 

27 is represented in the computer memory 102. At each scale, the applied image is divided into 

28 multiple, possibly overlapping sub-images based on a current window size. At each window, the 

29 processor 1 14 may attempt to classify the enclosed image pattern as being either a face or not a 

30 face. Each time a face window pattern is found, the processor 1 14 may report a face at the 
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1 window location, and the scale as given by the current window size. Multiple scales may be 

2 handled by examining and classifying windows of different sizes or by working with fixed sized 

3 window patterns on scaled versions of the image. Accordingly, in an image where people are 

4 scattered so there are faces of different sizes, the face detection algorithm, using the processor 

5 114, may find every face in the image. 

6 Although the image enhancement method using face detection is described using the face 

7 detection algorithms described above, one skilled in the art will appreciate that other face 

8 detection methods may be used in connection with the image enhancement, 

9 After the faces are detected and located in the image, the image enhancement method may 
1 0 automatically modify the image using, for example, mapping techniques, so that the image may 

pi 1 have preferred appearances, i.e., with more appealing lightness, contrast, and/or color levels, for 

JSJ12 example, and without any red eye artifact. 

^J 1 3 A t least one study has shown that people prefer to look at images, such as photographs 

UT114 and digital images, with certain levels of lightness and contrast, i.e., there are desirable levels for 

pi 5 a mean value and/or a variation value, such as a standard deviation, of the pixel values in the face 

^16 region. Using, for example, mapping techniques, the image enhancement method may modify 

Mill an image so that an output of the mapping may produce the image with the desirable levels for 

HI 18 the mean value and/or the standard deviation of the pixels in the face region. 

9 Lightness level in a color image is a component of the image that lends the perception of 

20 brightness. The image enhancement method will be described with respect to color images; 

21 however, one skilled in the art will appreciate that the method may equally be applied for 

22 processing monochrome images, as well as images represented with other color schemes, for 

23 example, sepia tone. 

24 An embodiment of the image enhancement method may add or subtract a fixed amount 

25 to the lightness component of each pixel in the image. Adding may lead to a brighter image, 

26 while subtracting may lead to a darker image. The processor 1 1 4 may select the fixed amount 

27 to be added or subtracted to produce an image with a target mean lightness level of the pixels in 

28 the face region. 

29 For example, x^may be the face pixels in an input image, where the symbol/represents 

30 a set of pixel locations recognized as being part of the face regions identified by the face 



HP No. 10006299-1 



1 detection algorithm. Suppose the mean of ay is rn x , and a transformation is preferred to ensure 

2 the mean of the face pixels in an output image is m r The pixels in the output image may be 

3 denoted with the letter y. In this example, the fact that pixel values usually have maximal and 

4 minimal levels, for example, 0 and 255, is ignored. In other words, "clipping" is ignored. The 

5 lightness transformation may use the following formula: y=x+T, where T=m t -m x . Since the 

6 average of x f is m x9 the average of y is m=m+m r m=m r Figure 2(a) illustrates the lightness 

7 transformation. 

8 Another embodiment of the image enhancement method may keep the mean of the 

9 lightness of the face pixels the same, and modify the standard deviation of the lightness of the 

10 face pixels with a fixed multiplicative factor. Similarly, the processor 114 may select the 

1 1 multiplicative factor that yields the desired level of variation. Following the notation of the 

12 above example, and again ignoring "clipping", the standard deviation of the face pixels in an 

1 3 input image may be written as a x . A target standard deviation may be referred to as a r The 



r- K 

1 4 contrast transformation may use the following formula: j=7x+(l -T)m x , where J- — J — j . This 

1 5 contrast transformation ensures that an output image may have the target standard deviation a t . 

16 Figure 2(b) illustrates the contrast transformation. 

1 7 Even though the image enhancement method is described using the mapping technique 

1 8 described above, one skilled in the art will appreciate that other image enhancement techniques, 

1 9 which work by modifying lightness, contrast, and/or color levels, may be utilized in connection 

20 with the face detection mechanism. 

2 1 The face detection algorithms described above typically further indicates the location of 

22 certain components of faces in an image, for example, eyes. Accordingly, the image 

23 enhancement method may further automatically reduce or remove any red eye artifact without 

24 human involvement, by simply passing the location of the eyes to red eye removal softwares 

25 stored in the memory 102 or the secondary storage device 1 12. 

26 The red eye artifact is a common artifact found in a photograph of a person or animal, 

27 especially when a flashbulb without a preflash is used when taking the photograph. The red eye 
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1 artifact, typically appearing as a red spot or halo obscuring all or part of the pupil of each eye, 

2 is typically produced when the pupil is sufficiently dilated to allow a noticeable amount of light 

3 from a source light to reflect off the back of the eye. In humans, the reflection is typically a 

4 reddish color or other colors. 

5 The image enhancement method may, after locating the eyes in the image, automatically 

6 determine if there is any red eye artifact in an image, and if yes, reduce or remove the red eye 

7 artifact from the human face without user interaction using the red eye removal technique. The 

8 red eye artifact may be reduced or removed by, for example, removing the redness in the eyes, 

9 making the eyes dark, or both. The red eye removal technique, traditionally requiring human 
1 0 involvement in clicking on the location in the image where the eyes are, is a well known digital 

Cjl image process. 

a 2 An example of a red eye removal technique is described in U.S. Patent No. 6,016,354, 

3 3 issued to Lin et. aL, entitled "Apparatus and a Method for Reducing Red-Eye in a Digital Image," 

j|| 4 which is incorporated herein by reference. U.S. Patent No. 6,016,354 discloses an apparatus and 

Cl5 method for editing a digital color image to remove discoloration of the image, known as a "red 

gl 6 eye" effect, by parsing the discoloration into regions and re-coloring the area of the discoloration 

fh 7 based on the attributes of the discoloration. The editing process automatically creates a bitmap 

Ul8 that is a correction image, which is composited with the source image or a copy of it and 

Cl9 displayed as the source image with the red eye artifact corrected. 

20 One skilled in the art will appreciate that other techniques for reducing or removing a red 

2 1 eye artifact may be used in connection with the image enhancement method using face detection 

22 to produce an enhanced image. After the image has been modified and enhanced, the image may 

23 be outputted through the output device 108 or the display device 1 10. 

24 Figure 3 is a flow chart of an exemplary image enhancement method using face detection. 

25 This method may be implemented, for example, in software modules for execution by processor 

26 114. After an image, such as a color photograph or a digital image, is inputted into a processor 

27 114, step 3 1 0, face detection algorithms may be used to automatically detect and locate human 

28 faces in the image, step 320. The face detection algorithms may also locate eyes in the human 

29 faces automatically for red eye reduction or removal, step 330. Next, image enhancement 

30 techniques may be used to automatically modify the image so that human faces may have 
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1 preferred appearances, step 340. The image enhancement may include enhancing lightness 

2 levels, step 342, enhancing contrast levels, step 344, enhancing color levels of the human faces, 

3 step 346, or enhancing other aspects of the image, step 348, to make the faces more appealing. 

4 The image enhancement technique may use mapping technique to process the image, step 350, 

5 i.e., determine mapping required to produce a more appealing image, so that when the mapping 

6 is completed, an output of the mapping may produce an image with the mean value and/or the 

7 standard deviation in the face regions achieving certain preferred target levels. The mapping may 

8 modify the faces alone or may modify the entire image. Finally, if any red eye artifact is 

9 determined to exist, step 360, the image enhancement method may automatically reduce or 

1 0 remove the red eye artifact from the faces, step 370. After the image is modified and enhanced, 

1 1 the image may be outputted through the output device 108 or the display device 110. 

12 While the image enhancement method has been described in connection with an 

1 3 exemplary embodiment, it will be understood that many modifications in light of these teachings 

14 will be readily apparent to those skilled in the art, and this application is intended to cover any 

1 5 variations thereof. 
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